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ABSTRACT 

A battery of 10 intelligence tests was administered 
to 22 subjects under standard conditions. After the testing session 
the degree of perceived similarity between 5 tests of the battery was 
to be estimated with regard to: (1) the kind of intellectual activity 
required by the tests, and (b) difficulty. Estimated qualitative 
similarity, according to (1) above was found to be a simple function 
of intertest correlation as determined from another group of 128 
subjects. A correlation of 0.48 was found between estimates of 
qualitative simularity and similarity with regard to perceived 
difficulty and a correlation of 0*40 between the latter similarity 
variable and intertest correlation. , When removing the effects of 
similarity with regard to perceived difficulty on the correlation 
between intertest correlation and estimated qualitative similarity 
(r2=0.79), a partial coefficient of correlation pf 0.74 was yielded, 
indicating that perceived difficulty accounts for only 12 percent of 
the association of 0.79 present. (Author) 
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EXPERIENCED INTELLECTUAL ACTIVITY AND PERCEIVED 
DIFFICULTY OF INTELLIGENCE TESTS* 



Bratfisch, O. Experienced intellectual activity and per- 
ceive J difficulty of intelligence tests. Reports from the In- 
stitute of Applied Psychology, the University of Stockholm. 
1972. No«30« - A battery of 10 intelligence tests was 
administered to 22 subjects under standard conditions. 
After the testing session the degree of perceived simi- 
larity between 5 tests of the battery was to be estimated 
with regard to (a) the kind of intellectual activity required 
by the tests and (b) difficulty. Estimated qualitative 
similarity (according to (a) above) was found to be a 
simple function of intertest correlation as determined 
from another group of 128 subjects. A correlation of 
0.48 was found between estimates of qualitative similarity 
and similarity with regard to perceived difficulty and a 
correlation of 0.40 between the latter similarity variable 
and intertest correlation. When removing the effects of 
similarity with regard to perceived difficulty on the 
correlation between intertest correlation and estimated 
qualitative similarity (r = 0.79), a partial coefficient 
of correlation of 0.74 was yielded, indicating that per* 
ceived difficulty accounts for only 12 percent of the 
association of 0.79 present* 

Int roduct ion 

In recent years a number of studies have been concerned with the 
perception of various performance activities. In general, the primary 
interest was in the relation between some defined aspect of a per« 
formance activity as perceived by the performing subjects themselves 
and corresponding "objective** measurements being of a physical, 
physiological or psychological (though non -perceptive) nature. 

Probably the first serious attempts to study performance on the 
perceptual level were undertaken in the area of physical work (e»g* 
Borg, 1962; Borg & Dnhlstrdm, 1959, I960), in which field also 
the practical impacts of the obtained findings have become obvious; 
for a survey see e.g. Borg,( 1970, 1971). Later on the perception of 
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performance activities in fields like motorskill (Bratfisch, Dornfcf 
& Borg, 1970), immediate memory (Borg, Bratfisch & DorniV^, 
1971a) and visual perception (Borg, Bracfisch & Dornfci', 1971b) were 
studied. More recently a number of investigations have been con- 
cerned v/ith the perception of intellectual performance. The majority 
of the latter kind olstudies have dealt with perceived difficulty (Borg, 
1966, 1969; Backman & Wedman, 1971; Munz & Jacobs, 1971; 
Bratfisch. Borg & Dornitf, 1972; Bratfisch, Dorni>5 & Borg, 1972), 
while some others were interested in the dimensionality of intellectual 
performance based on the perception of the intellectual activities 
involved (Bratfisch & Ekman, 1969; Bratfisch, 1971). 

The present study is once more concerned with the perception 
of intellectual performance and can be regarded as a continuation of 
the previous investigations on the topic combining both features men- 
tioned, namely the perceived difficulty as well as the perceived quality 
(intellectual activity involved) of intellectual tasks. 

A very close correspondence between subjective intelligence 
factors, based on the perception of similarity between tasks with 
regard to the intellectual activities involved, and objective performance 
factors, extracted from analyses of correlation data, was found earlier 
(Bratfisch & Ekman, op.cit., Bratfisch, op,cit.). It could also be 
demonstrated there that the perception of qualitative similarity between 
the tasks was not due to similarity with regard to difficulty as ex- 
pressed by differences between performance scores. However, it 
might be due to the perception of the difficulty of the tasks. Thus, the 
major question to be illuminateu in the present study was: what is 
the relation between perceived qualitative similarity and similarity 
with regard to perceived difficulty of intellectual performance ac- 
tivities? As the investigation of this question involves a-comptex 
stimulus situation measurements concerning the subjects' perception 
of uncertainty about their own estimates of similarity were considered 
to be of additional interest. 



Method 

The stimulus situation 

Ten factor tests from a standardized intelligence test battery 
(the "Delta Battery" of the Institute of Applied Psychology; manual, 
1971) were u^ed as stimuli in the experiment. All the tests were 
conventional in character and may, hence, be characterized by names 
only. In the following presentation the tests are arranged accordmg 
to the factors they represent: the denotion of factors follows the 
system introduced by Thurstone (1938) and Thurstone & Thurstone 
(1941)^ 





Test 


Factor 




(I) 


*j y 11 v/iiy Alia 


Verbal comprehension 


(V) 


(Z) 




tf 






*■ 


Perceptual speed 




(4) 


i T* 1 9 I V% % % ^% ^ 

jiueiiLicax niimuers 


ft 






JN^ultiplicEt ions 


Numerical facility 


(N) 




iviixea cumpuiat ions 


ft 




(7) 


Number series 


Reasoning ability 
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Matrices 
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Levers 


Spatial ability 
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(10) 


Surface development 
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The tests were administered to the subjects under standard con-» 
ditions. The testing session served the purpose, among other things, 
of making the subjects thoroughly familiar with the tests and thus 
providing them with a basis for judging what kind of intellectual per- 
formance was required by any particular test and what degree of 
difficulty they might have. 

Estimation of test difficulty 

After the testing session the subjects were asked to estimate the 
perceived difficulty of the various tests on a symmetrical scale of 
nine categories with verbal expression labels as follows: 1 • very, 
very easy; 2 - very easy; 3 - easy; 4 - rather easy; 5 - neither 
easy nor difficult; 6 * rather difficult; 7 - difficult; 8 - very 
difficult; 9 - very, very difficult. They were instructed to base their 
judgements on their spontaneous, entirely personal impression with- 
out considering what other people might experience in this connection* 

To study estimated test difficulty was not a primary aim of the 
present investigation* However, as estimates of test difficulty were 
readily obtainable, they were included in the experimental proceedings* 
With reference to the positive experience of category estimation 
made in earlier studies on perceived difficulty of individual test 
items (e.g. Bratfisch, Borg & Dornfcf , 1972) this method was pre- 
ferred to other non time-consuming methods like e.g. magnitude 
estimation* 

Similarity estimation 

After the estimates of test difficulty subjects were given a break 
of about one hour before proceeding with the second part of the ex- 
periment. Five tests, one for each of the factors V, P, N, R, and S, 
sampled from the above described battery, served as stimuli in this 
part of the experiment, where similarity between the tests was to be 
estimated from two aspects: similarity with regard to perceived 
difficulty and similarity with regard to the intellectual ability per- 
ceived to be involved when dealing v/ith the various tasks. The five 
tests were: 1. Cpposites, 2. Identical letters, 3. Mixed computa- 
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tions, 4, Nialiices, ajid b. Surface development, A redurcior* of 
stimuli turned out to be necessary as che usage of all 10 tests as 
stimuli instead of 5 wcmic? hrr-r^ prolonged the experiment ty about 
11/2 hci.i'^, v.'i::ch was impossible to carry out in practice; the 
experiment as it was performed actually lasted around 7 hours. 

The names of the te = t£ were presented in all 10 possible pairs 
on printed answer sheets. The order of pairs was randomized 
differently for the two kinds of similarity estimates to be given. 

In the ''difficulty" condition the subjects were instructed to con- 
sider each pair of tests in turn and to estimate the decree of similar- 
ity between tests with regard to perceived difficulty. They were again 
asked to base their judgements on their immediate, entirely personal 
impression without considering what other people might perceive. 
The estimates of similarity were to be given on a percentage scale, 
too representing identity and 0 denoting no similarity at all. The 
percept ^of each pair perceived as most difficult was to be pointed out 
in order to be able to erect a scale of subjective difficulty. The sub- 
jects were given a number of training trials* with the remaining 5 
tests not selected, in order to ensure a correct understanding of the 
instructions; the names of the ten tests together with a sample item 
from each test were placed in front of the subjects as an aid to 
memory. In addition, to get a hint of the subjects ^ momentary im- 
certainty concerning the adequacy of the given estimates, so called 
interval estimates were to be delivered (see Hallsten, 197ia). This 
implies that subjects are asked to report the smallest interval within 
which their percept is located with a certain degree of personal 
credibility. In the present case this was done by asking the subjects 
to specify the interval that they were almost definitely certain in- 
cluded their percept of similarity between the pairs of stimuli. Thus» 
three estimates were given for each pair of stimuli: a ''point'* - 
similarity estimate as well as an estimate of the lowest and an es- 
timate of the highest figure connected with a certain percept. 

In the "quality" condition the subjects were instructed to con* 
sider again each pair of tests in turn and to estimate the degree of 
similarity between the kinds of intellectual ability involved when 
dealing with the tasks of these tests* They were carefully instructed 
to base their judgements entirely on qualitative similarity between 
the tests, disregarding possible perceived differences in difficulty. 
Otherwise the procedure was the same as in the difficulty condition. 

One half of the subjects estimated qualitative similarity first 
and the other half similarity with regard to difficulty. Both conditions 
of similarity estimation were carried out individually whereas the 
testing session and the direct estimates of difficulty were performed 
in groups of 3-*6 persons. 

Subjects 

22 subjects participated in the experiment. All of them were 
vocational guidance clients of the Institute of Applied Psychology, 
and the experiment was conducted within the regular guidance pro- 
gram. The age of the subjects ranged from 18 - 47 years, the median 
age being 25 years^ There were 12 male and 10 female subjects. All 
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of them had elementary (9 years) schooling only, A homogeneous 
group of this composition was preferred for two reasons: the subjects 
should be unsophisticated in intelligence theory, and they should 
represent a population for which the tests were primarily constructed 
and standardized. Moreover, to qualify as a subject a minimum 
averaged test performance of stanine 5 turned out to be a necessary 
requiriement as the procedure of similarity estimation in the two 
outlined respects was a rather demanding task in itself. 

Results 

Similarity estimation, interval estimates, coefficients of correlation 

Medians and semi-interquartile ranges were computed for the 
individual "point" -estimates of similarity of each pair of tests due 
to some skewed distributions. These statistics are shown in Tables 
lA and IB for the quality condition and the perceived difficulty 
condition, respectively, together with the corresponding statistics 
concerning the interval estimates, i.e. the lowest and the highest 
figure subjects could still connect with a given percept. Moreover, 
arithmetic means of as well as ratios between the median lower and 
the median upper estimates of each pair of stimuli are given in these 
tables. Finally, coefficients of intertest correlation, obtained from 
another group of 128 vocational guidance clients, are seen in Tables 
lA and IB. In the following, "point" -estimates will be referred to 
just as "similarity estimates", while the two interval estimates will 
be called "upper" and "lower" estimates. 

Table lA. Medians (Mdn) and semi-interquartile ranges (Q) computed 
from experimental estimates. The data refer to similarity 
with regard to intellectual ability involved when dealing 
with the various tests. L = median lower estimates, U = 
median upper estimates, S = median similarity estimates; 
Op = Opposites, IL = Identical letters, MC = Mixed 
computations. Ma = Matrices, SuD = Surface develop- 
ment, r = intertest correlation. 
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■tr 


Mdn^^ 
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Mdn^/Mdfiy 
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Op - IL 


28.5 


28.5 


37. S 


35.0 


33.00 


30.0 


29.0 


0.76 


0.40 


Op - MC 


1^.0 


24.5 


28.5 


32. S 


21.50 


25.0 


22.5 


0.53 


0. 24 


Op - Ma 


56.0 


20.0 


S5.0 


22.5 


45.50 


45.0 


22.5 


0.65 


0.47 


Op - SuD 


12.5 


12.5 


22.5 


19.0 


17. 50 


17.5 


14. D 


0..56 


0. 27 


IL - MC 


27.5 


34.5 


51.0 


35.0 


39. 25 


40.0 


36.5 


0.54 


0.32 


IL * Ma 


35.5 


23.5 


52.5 


35.0 


43. 75 


50.0 


27.5 


0.67 


0. 51 


IL - SuD 


5.0 


10.0 


22.5 


20.0 


13.75 


15.0 


13.5 


0. 22 


0.23 


MC - Ma 


15.0 


14.0 


28. S 


22.5 


21. 75 


22.5 


21.0 


0.53 


0. 32 


MC - SuD 


15.0 


14.5 


26.5 


28.5 


20. 75 


19.5 


23.5 


0.57 


0.13 


Ma • SuD 


44.0 


35.0 


62.5 


25.0 


53.25 


50.0 


30.5 


0.70 


0.36 
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Table IB. Medians (Mdn) and semi-interquartile ranges (Q) computed 
from experimental estimates. The data refer to similarity 
with regard to perceived difficulty. For an explanation of 
the abbreviations used see table lA. 
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Op - IL 


40.0 


27. 5 


62.5 


30.0 


51. 23 


50. 0 


30.0 


0.64 


0.40 


Op - MC 


46.5 


25.0 


60.0 


20.0 


53. 25 


53.0 


25.0 


0.78 


0*24 


Op - Ma 


52.5 


20.0 


75.0 


17.5 


63.75 


65.0 


17.5 


0.70 


0.47 


Op - SuD 


35.0 


15.0 


6C.0 


15.0 


47. 50 


50.0 


iS.O 


0.58 


0.27 


IL - MC 


60.0 


20.0 


81.0 


20.0 


70. 50 


75.0 


20.6 


0.74 


0.32 


IL - Ma 


45.0 


20.0 


60.0 


17. 5 


52. 50 


50.0 


25.0 


0.75 


0.51 


IL - SuD 


32.5 


17.5 


60.0 


16.0 


46. 25 


46.0 


17.5 


0.54 


0.23 


MC - Ma 


43.5 


17.5 


60.0 


15.0 


51.75 


50.0 , 


15.0 


0.73 


0.32 


MC - SuD 


28.5 


12.5 


51.0 


15.0 


39. 75 


40.0 


12.5 


0.56 


0.13 


Ma - SuD 


50.0 


20.0 


70.0 


17.5 


60.00 


50.0 


17.5 


0.71 


0.36 



Scales of estimated test difficulty, test performance 

Table 2 shows tmder the heading "direct scaling*^ means of the 
experimental estimates of the difficulty of the 10 tests applied as 
stimuli obtained by the category scaling. For comparative purposes 
a scale of subjective difficulty was constructed, assuming Thurstone's 
Case V, on the basis of rank order information obtained as part of 
the similarity estimates concerning the "difficulty" conHition. This 
scale is also seen in Table 2 under the heading "indirf . scaling" 
together with the subjects' mean performance scores expressed in 
stanine units obtained by the standardization group (Delta Battery, 
1 ^7 1 ^ » 



Table 2. Scales of subjective difficulty, constructed from experimen- 
tal data, test scores. 
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Synonyms 


3.77 




6.45 


Opposites 


4.45 


4.19 


7.64 


Identical letters 


2.77 


2.40 


4.95 


Identical numbers 


2.36 
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6.41 


Mult iplicat ion s 


4.18 




5.18 


Mixed computations 


4.05 


3.60 


5.55 


Number series 


5.32 




5.50 


Matrices 


4.91 


4.91 


5.73 


Levers 
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5.73 


Surface development 


6.18 


5.84 


5.59 
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Uncertainty in similarity estimation 

From both Tables lA and IB it can be seen that the inter- 
individual momentary precision of the estimates of similarity as 
expressed by the ratio between the median lower and the median 
upper estimates of each pair of stimuli is in most cases quite high. 
Such statistics may be regarded as an expression for the subjects' 
own uncertainty in connection with a given percept assuming equality 
of the individual density functions and equality of the individual levels 
of credibility behind the estimates of similarity. A ratio of 1.00 
would mean that the subjects on an average felt positive about the 
adequacy of the figure ascribed to a given pair of stimuli; a decreasing 
degree of precision is expressed by decreasing ratios. The relatively 
high degree of precision throughout in the estimates obtained is in- 
teresting in the light of results from studies on perceived confidence 
limits of estimates of physically simple stimuli like loudness and 
line length (e.g. Hallsten, 1971b', 1971c). In these studies a somewhat 
lower degree of precision in estimates was found than in the present 
study which certainly can be said to have involved a complex stimulation. 
On the other hand, however, subjects might just have calculated the 
limits instead of expressing their experience numerically. When 
comparing the arithmetic means of the interval estimates (Mdn. + 
Mdn^/2) with the corresponding median similarity estimates 
(S) in Tables lA and IB a very close conformity can be noticed; 
the two kinds of statistics differ but little. 

Another way of analyzing the inter -individual uncertainty would 
be to examine the relation betwer * the inter -individual variability 
in the similarity estimates and the corresponding central measure- 
ments. This relation has earlier been found to be described by an 
inverse U-shaped trend, probably of elliptic character (Ekman & 
Kiinnapas, 1969) . The present data, however, are not suitable for 
revealing such a trend. This is due partly to the few observations 
available (usually a considerable scatter around tho fitted curve is 
found) and partly to the fact that the estimates range only from 15 
to 50 with regard to the quality condition and from 40 to 75 as far 
as the quantity condition is concerned. With a range of 15 - 50 we 
would expect inter -individual varinHility to increase with increasing 
medians, which by and large is the case (cf Table lA), and with a 
range of 40 - 75 we would expect the differences in variability between 
estimates just not to be too great; the semi -interquartile ranges of 
the similarity estimates seen in Table IB seem on the whole not to 
be contradictory to this expectation. 

If the two above outlined indices of uncertainty would be ex- 
pressive for the same underlying phenomena they would be negatively 
related to each other (Hallsten, 1971a). However, as can be seen in 
Tables lA and IB, this is not the case in the present investigation. 
This problem is worth studying in more purely methodoligcal research. 

Similarity and correlation 

The medians of the similarity estimates with regard to the 
quality condition, divided by 100, have been plotted against co- 
efficients of correlation in Fig, lA (cf Table lA). For comparison, 
the same kind of data plotted from an earlier study (Bratfisch & 
Ekman, 1969) is seen in Fig. IB. In that study qualitative similarity 
between ten tests (identical with those of the present study) was to 
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be estimated Thus 45 pairs of stimuli were judged compared to 10 
m the present study. The 10 points in Fig. IB corresponding to those 

t^en?;f 'trK^r^*^ ^ ^'^^'^ I" study the 

trend of data could be brought out more clearly by means ox an 

averaging procedure; this trend is seen in Fig. IC. 




0X> 0.1 0.2 0.3 0.4 0.5 



-0.2 O.O a2 0.4 0.6 0.8 
- Coefficient of correlation (r) 



0.0 

-0, 



2 0.0 0.2 0.4 0.6 0.8 



Fig. 1. Qualitative similarity estimate as a function of coefficient of 
correlation. Diagram A refers to data from the present ex- 
periment, Diagrams B and C show the results from an earlier 
study (Bratfisch & Ekman, 1969). The curves drawn in Dia- 
grams A and C represent Equation 1. 

The trend of the data in Fig. lA, i. e. the relation between 
qualitative similarity estimates and corresponding coefficients of 
correlation, may be desicribed by the equation 

2 

s « a + br ^jj 

where s refers to qualitative similarity and r to correlation; a and b 
are empirical constants. This particular mathematical function was 
applied to the present data with reference to resiults from two earlier 
studies on the present topic (Bratfisch & Ekman", 1969; Bratfisch, 
1971) and from one partly analogous study concerning personality 
traits (Magnus son & Ekman, 1970). On the other hand it is clear 
that alternative functions could have been fitted to the present data 
(which was also the case in the earlier studies mentioned). In the 
present state of affairs no theoretical interpretation can be offered 
for the equation actually applied nor could that be done for any other 
function which could have been chosen. Nevertheless, it is now the 
fourth time equation (1) is approximately descriptive for experimental 
data; that a simple, regular relation exists between correlation and 
similarity ?eems to be clear and maybe we are on our way towards 
an empirical law. 

Estimates of qualitative similarity, expressing the subjects' 
OM-n perception of overlap between stimuli, and correlation co- 
efficients, indicating the extent to which performance tends to co- 
vary over subjects, are basically of a different nature. As has been 
said, no simple model is available to account for the obtained relation 
between the two variables. However, estimates of qualitative similar- 
ity might be influenced by the perception of the difficulty of the tests, 
though they did not appear to be related to performance scores (see 
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Bratfis.ch & Ekman, 1969; B^tfisch, 1971). One of the major topics 
of the present study was to inTbstigate this problem. Median estimates 
of qualitative similarity have been plotted against median similarity 
estimates with regard to perceived difficulty in Fig. 2A (cf Tables 
lA and IB). Fig. 2B shows median similarity estimates with regard 
to perceived difficulty plotted against coefficients of correlation 
(cf Table IB). 



> 

•r- 

to 

•I- 4-> 



OH- 
4-> O 

4/1 to 

O 
c u 
to 

-a 4-> 



10 



0.8 



0.6 



0.4 



0.2 



o.qL^ 



o o° o 



r»0.48 



I.O f^ ■!■ 



0.1 0.2 0.3 0.4 0.5 
Median efttlmates of quatttattve fttmtlarlty 



0.8 



0.6 ' 



0.4 



0.2 



0.0 



! I 



B 



o o o o o 



r-o.4o 



0.1 0.2 0.3 0.4 0.5 
Coefficients of correlation 



Fig. 2, Similarity estimates with regard to difficulty as related to 
qualitative similarity estimates (Diagram A) and co- 
efficients of correlation (Diagram B). 

No curved trend can be noticed in either of the graphs of Fig. 2; 
hence, coefficients of correlation have been computed. They are 
0.48 for the data in Fig. 2A and 0.40 for the data in Fig. 2B. Both 
coefficients are of course not significant, due to the small number 
of observations, but turned out to be by and large descriptive for 
individual data. Nevertheless, if we accept these statistics at their 
face value, interesting additional computations can be carried out. 

The relation between estimates of qualitative similarity and 
intertest correlation, previously described by Equation (1), can also 
be stated in correlational terms, the coefficient of correlation then 
being 0.79. Now, 'knowing the correlation between intertest correla- 
tion and similarity with regard to difficulty (0.40) as well as the 
correlation between estimates of qualitative similarity and similarity 
with regard to difficulty (0.48) we can quantitatively express the 
proportion overlap which results from the effect of .similarity with 
regard to difficulty of the correlation of 0.79 between intertest 
correlation and estimates of qualitative similarity. We simply 
remove the effects of similarity with regard to difficulty by calcu- 
lating the partial coefficient of correlation, which turns out to be 
0.74. The proportion overlap which results from^^*^® effec^ of 
similarity with regard to difficulty is then (0,79) - (0.74) = 0.0765, 
We could also say that the percentage of the total covariation between 
intertest correlation and estimates of qualitative similarity present 
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resulting from the effect of similarity with regard to difficulty is 
(0.0765/0.79 ) = 12 percent; the remaining 88 percent of the 
association results from other factors* Thus perception of the 
difficulty of the tests seems to have had but minor effects on the 
obtained relation between intertest correlation and qualitative 
similarity* 

The estimates of similarity have also been related to performance 
scores in the same way as has been done previously {Ekman & Brat- 
fisch, 1969; Bratfisch, 1970) where it was found that judged qualita- 
tive similarity was not affected by the individual difficulty of the 
tests; however, it would be plausible if estimates of similarity with 
regard to difficulty were affected. For each subject and each pair of 
tests, the difference (disregarding sign) between the two standard 
scores on the test was calculated and *elated to estimates of similar- 
ity* Table 2A shows such data for es<"^ma ' of qualitative similarity 
and Table 2B shows data for estima »ilarity with regard to 

difficulty. 

Table 2A. Frequency of individual estimates of qualitative similarity 
as related to individual differences between standard test 
scores. 



Standard score 








Estimates of qualitative 


similarity 








difference 


0.05 


0. 15 


0. 25 


0.35 


0.45 


0.55 


0.65 


0.75 


0.85 


0.95 


N 


0 


3 


3 


2 


3 


3 


4 


2 


7 


t 


3 


31 


1 


19 


10 


7 


3 


1 


3 


4 


2 


6 


2 


57 


2 


15 




5 


2 


4 


8 


7 


1 


5 


4 


54 


3 


8 




6 


1 


4 


2 


3 


4 


0 


2 


33 


4 


6 




4 


2 


4 


5 


2 




0 


2 


28 


5 


3 












1 




1 


1 


8 




4 




















7 


7 


1 




















2 


8 


0 


0 


















0 




59 


28 


?4 




16 


22 


19 


14 


13 


14 


220 



Table 2B. Frequency of individual estimates of similarity with regard 
to difficulty as related to individual differences between 
standard test scores. 



Standard score 
difference 


0 .05 


Estimates of similarity with regard to difficulty 
0. 15 0 .25 0.35 0.45 0. 55 0.65 0. 75 0.85 


0.95 


N 


0 






6 


2 


5 


2 


4 


3 


7 


32 


1 


1 




4 


7 


9 


It 


6 


5 


6 


55 


2 






8 


7 


9 


4 


6 


2 


6 


53 


3 


1 




2 


1 


7 


5 


5 


2 


4 


32 


4 


1 




2 


3 


4 


2 


2 


3 


2 


28 


5 


1 






\ 


1 




1 




1 


8 


6 






1 


2 


2 




1 


1 




9 


7 










1 










2 


a 




















1 




4 


12 29 


23 


23 


38 


24 


25 


16 


26 


220 
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Table 2A reflects the same results that have been obtained pre- 
viously; there is practically no trend of similarity decreasing with 
increasing performance difference. Surprisingly enough there is only 
a very weak tendency for estimates of similarity with regard to 
perceived difficulty to covary with differences in performance scores; 
the data in Table 2B are nearly as randomly distributed as the data 
in Table 2A. 

Estimated test difficulty 

When comparing the scale values of the five tests applied in 
indirect scaling with the corresponding scale values obtained by 
direct scaling in Table 2,a very close corre.-j^>G4idence can be noticed* 
In fact, the coefficient of correlation between the two sets of data is 
0.99. Hence it may be concluded that the category scaling applied 
yielded an interval--scale provided that the assumptions on which 
Thurstone' s Case V rests are. fulfilled. 

When comparing the mean estimates of test difficulty with the 
corresponding mean test performance (cf Table 2) no tendency can be 
noticed that estimated difficulty decreases with increasing test scores 
The restriction of the range of the performance continuum (to qualify 
as a subject a minimum average performance .score of stanine 5 
was required) is probably the main reason for that. Moreover, the 
size of the experimental group was too small to distinguish between 
subjects scoring relatively low and relatively high on a certain test, 
which would have been the way to illuminate the relation between 
estimated test difficulty and test performance* For such an analysis 
a larger group of, say, at least 50 subjects heterogeneous with 
respect to performance will be needed* 

Discussion 

The primary aim of the present investigation was to illuminate 
the relation between estimates of perceived qualitative similarity 
and estimates of similarity with regard to perceived difficulty of 
intelligence tests. This aim was guided by the endeavour to unveil 
the influence of similarity with regard to difficulty on the perception 
of qualitative similarity, which had been found to be closely related 
to intertest correlation. A relatively low correlation between the two 
subjective variables of C.48 was found, which is statistically not 
significant. Howeveir, when taking this correlation at its face value 
the influence of similarity with regard to difficulty on the relation 
between estimates of qualitative similarity and intertest correlation, 
based on performance, cciUd be isolated. The percentage of the 
total association betwe^in intertest correlation and estimates of 
qualitative similarity present (r = 0.79) resulting from the effect 
of similarity with regard to perceived difficulty turned out to be as 
low as 1 2 per cent. 

The findings of two earlier studies on this topic, namely the 
close correspondence between estimates of qualitative similarity and 
intertest correlation expressed by a power function with an exponent 
of 2.0, could in a way be c^firmed in the present study. The in- 
fluence of perceived difficulty appears to have only minor effects on 
the obtained relation between these two psychologically fundamentally 
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different variables; however, there is still no simple model available 
to account for it. Generally speaking it can be stated relatively 
accurately that such common elements that produce positive per- 
formance correlations also account for the overlap expressed by the 
estimates of qualitative similarity. 

A few words should be said about the dimensionality of the es- 
timates of qualitative similarity. In the two earlier studies on this 
topic (Bratfisch & Ekman, 1969; Bratfisch, 1971) a very close agree- 
ment between the ''subjective'* factor matrix, based on estimates of 
qualitative similarity and the ^'objective" factor matrix, based on 
inter-test correlations, was obtained. With only five tests available 
in the present study it would be psychologically less significant to 
apply factor analysis to data. However, when comparing estimates 
of qualitative similarity of the present study with the corresponding 
ones of the two studies mentioned a high conformity could be noticed. 
Thus, it may be concluded that the underlying dimensionality of the 
estimates of qualitative similarity of the present study with high 
probability is identical with the one obtained in the earlier studies. 

The question whether or not estimates of difficulty of a whole 
intelligence test also are linearly related to objective performance as 
are estimates of difficulty of individual tasks, which was found in 
earlier studies, could not be answered properly through the present 
study. This is probably due to the restriction of th4 range of the 
performance continuum present. In spite of the fact that subjects could 
not be differentiated according to performance it is interesting to 
notice that subjects did perceive pronounced differences in the diffi- 
culty of the tests. Tests of perceptual speed were judged as relatively 
most easy and tests of spatial ability were estimated as relatively most 
difficult. It would be worthwhile to study the problem of the perception 
of total test difficulty in future research with larger and more hetero- 
geneous groups of subjects. 



References 

Backman, J. & Wedman, I. Differentiell poilngsSttning av flervals- 
f rigor med viktsystem grundade pi prestations - och 
perceptionsmStningar. Pedagogiska rapporter, Universi- 
tetet och LSrarhOgskolan i Umei, nr 21, 1971 (in Swedish). 

Borg, G. Physical performance and perceived exertion. Lund (Sweden): 
Gleerup; 1962. 

Borg, G. Nigra psykof ysiska tillSmpningar. Nordisk Psykologi, 
1966, 18, 300-315 (in Swedish). 

Borg, G. The three effort continua in physical work. Proceedings of 
the XVI International Congress of Applied Psychology, 
Amsterdam, 1968, 394-397. 

i 

Borg, G. Upplevd svSrighet hos uppgifter i Ravens matriser. Un- 
published manuscript, the Institute of Applied Psychology, 
the University of Stockholm, 1969 (in Swedish). 



ERIC 



Borg, G. Perceived exertion as an indicator of «omatic stress. 

Scandinavian Journal of Rehabilitati on Medicine, 1970, 
2-3, 92-98. 

Borg, G. Psychological and physiological studies of physical work. 

In T.W. Singleton, LG.Fox & D.Whitfield (Eds): Measure- 
ment of man at work. London: Taylor & FranciS, 1971, 
121-128. 

Borg, G. & Dahlstrom, H. Psykofysisk undersokning av arbete p& 
cykelergometer. Nordisk Medicin, 1959, 62, 1383 - 1386 
(in Swedish) . 

Borg, G. & DahlstrOm, H. The perceptior j-.. n.uscuLir work. Pub- 
lications of the UmeS Research Library, 5, I960, 1-27. 

Borg, G. k Forsling, S. A psychological studv on perceived difficulty. 
Educational and Psychological Kusaarch Bulletin, the 
University of Ume&, No. 1, 1964. 

Borg, G. & Forsling, S. En differentiell psykofysisk undersokning 
med ett prov pi fSrm&ga att utfora matematiska kvot- 
skattningar. Rapport fr&n pedagogisk-psykologiska 
institutionen, Ume& universitet, nr 5, 1965 (in Swedish). 

Borg, G., Bratfisch, O. & Dornftf, S. Perceived difficulty of an 
immediate memory task. Reports from the Institute of 
Applied Psychology, the University of Stockholm, 
No. 15, 1971 (a) 

borg, G., Bratfisch, O. & Dornftf, S. Perceived difficulty of a 

visual search task. Reports from the Institute of Applied 
Psychology, the University of Stockholm, No. 16, 1971 (b). 

Bratfisch, O. A further study on subjective and objective intelligence 
factors. Reports from the Institute of Applied Psychology, 
the University of Stockholm, No. 20, 1971. 

Bratfisch, O. & Ekman, G. Subjective and objective intelligence 
factors. Psychological Reports, 1969, 25, 607-620. 

Bratfisch, O., Borg, G. & Dornitf, S. Perceived item-difficulty in 
three tests ofintellectual performance capacity. Reports 
from the Institute of Applied Psychology, the University 
of Stockholm, No. 29, 1972. 

Bratfisch, O. , Dornii5, S. & Borg, G. Perceived difficulty of a motor- 
skill task as a function of training. Reports from the 
Institute of Applied Psychology, the University of Stock- 
holm, No. 11, 1970. 

Bratfisch, O., DorniV^, S. & Borg, G. Perceived difficulty of items 
in a test of reasoning ability. Reports from the Institute 
of Applied Psychology, the University of Stockholm, 
No. 28, 1972. 



Delta Battery (Delta batteriet). Manual. Skandinaviska Testf5rlag6t, 
Stockholm, 1971 (in Swedish) ♦ 



-14- 



Ekman, G. & Kiinnapas, T. The distribution function for similarity 
estimates. Reports from the Psychological Laboratories, 
the University of Stockholm, No. 286, 1969. 

Hallsten, L. Precision of stimulus -directed hypotheses in a mag- 
nitude estimation experiment. Reports from the Psycho- 
logical Laboratories, the University of Stockholm, 
No. 322, 1971 (a). 

Hallsten, L. Interrelations between simultaneous point and interval 
estimates under percept - and stimulus -directed in- 
structions. Reports from the Psychological Laboratories, 
the University of Stockholm, No. 323, 1971 (b). 

Hallsten, L. Point estimates, interval estimates and ratio scale 
demands in a ratio estimation experiment. Reports 
from the Psychological Laboratories, the University 
of Stockholm, No. 324, 1971 (c). 

Magnusson, D. & Ekman, G. A psychophysical approach to the 
study of personality traits. Multivariate Behavioral 
Research, 1970, 5, 255-274. 

Munz, D.C. & Jacobs, P.D. An evaluation of perceived item- 
difficulty sequencing in academic testing. British 
Journal of Educational Psychology, 1971, 44, 195-205. 

Thurstone, L.L. Primary mental abilities. Psychometrical 
Monographs, 1938, 1. 

Thurstone, L.L. & Thurstone, T.G. Factorial studies of intelli- 
gence. Chicago: University of Chicago Press, 1941. 
(Psychometric Monographs 2) 



ERIC 



u 
u 

05 



u 



0) 

I 

^< o 
g « ^ 

(« 4-» 'H 

> 0) > 

13 

••^ ^ y o 

O 4-1 « . 

tt> U 

•H o a; ^ 

° 6 

4^ o 

^ u A CO 



u 

u 
u 

CO 



ERIC 



1^ 



2 

U * 

^ o 
p. 



CO 



(0 

O 



13 >N 

(0 '"^ ^ 

4-» f« M-i > • rj 

in ^ 2 iS 

^ ^ 'tit 



13 



•2 y .ti c 



4J (0 

.3 *! 

13 tt> 

U fl 

• O 

o 



> *H o 

^ ^ 



H Li rti ;2 " 
- i tt) O c 

S iJ y 

cr 3 ^ 

<U E (u 



:^ 

^ 4J 

< rtJ 

4-» 17 

" 5 ? 

p ^< 

(0 (0 0) 
0) 



13 

^ M (0 

•H .J5 *Q 
•H O (U 

(0 d 13 ^ 
C d) 



13 
> 

u 

0) 



0) o 



(11 



13 
> 

U CO 
U 0) 



5) 4> 



U ^ 



»4 

0) (1) 
4^ >s 



:3 cro 



I 



43 ;3 
u u 
CO ;:2 

VH 

4^ JlI 



Us " 
^5 o 

o 2 ii 



tt) ,^ 'H 4-» 

^ S M 

^ (« U 4-» 

c ^ ^ 3 

o ^ c o 

♦H 4-» c *^ 

M *H 4J CO 

^ ^ ^ ,^ 5 

n o o M > 



O 13 
C C ^ 

«j g 5 

O M 

o 



o 

c c *^ ^ 

tt> O 13 

.J <^ 4J 

0) a> 0) 0) 



VH > CO 



O U 
O 



•5 i 



o 

^ o 

^ >^ s ^ 

^ " 6 .2 

r;;* 13 'H 4-» 

£ C CO 

t: 

O 0) o 

^3 « 

(T) (T) f) 



.i{ P. CO 

vTJ CO 

oJ «J 

O ^ OJ 

<^ *2 

•H *rj O 

«>.ii £ 

^ u 

iJ <^ 

>i « o 

>^ ^ o 

!3«- " 



•'S 2 

N 

^ O 

cr u 



c * 

o c 

u ^ 

>. P. 

4*> 
•H 

u o 



REPORTS SO FAR PUBLISHED IN THIS SERIES 



No. 1, 1970 Borg, G. : RELATIVE RESPONSE AND STIMULUS 
SCALES 

No. 2, 1970 Bratfisch, O. : TIME-ESTIMATIONS OF THE MAIN 
ACTIVITIES OF UNIVERSITY STUDENTS 

No. 3, 1970 Hosman, J., and Borg, G. : THE MEAN AND STANDARD 
DEVIATION OF CROSS-MODALITY MATCHES: A STUDY 
OF INDIVIDUAL SCALING BEHAVIOR 

No. 4, 1970. Borg, G. , Edstrom, C-G. , and Marklund, G. : A NEW 
METHOD TO DETERMINE THE EXPONENT FOR PER- 
CEIVED FORCE IN PHYSICAL WORK 

No. 5, 1970 Hosman, J. : THE FACTOR STRUCTURE OF MAGNI- 
TUDE PRODUCTIONS 

No. 6, 1970 Hosman, J. : THE DIMENSIONALITY OF CROSS-MO- 
DALITY MATCHES 

No. 7, 1970 Borg, G. , and Hosman, J.: THE METRIC PROPERTIES 
OF ADVERBS 

No. 8, 1970 Borg, G. , Edgren, B, , and Marklund, G. : A FLEXIBLE 
WORK TEST WITH A FEEDBACK SYSTEM GUIDING 
THE TEST COURSE 

No. 9, 1970 Domic, S. , Kiinnapas, T. , and Bratfisch, O. : SUBJEC- 
TIVE SIMILARITY AS A FUNCTION OF EXPOSURE 
TIME AND SHORT-TERM MEMORY 

No. 10, 1970 Borg, G. , Bratfisch, O, , and Domic, S. : ON PER- 
CEIVED DIFFICULTY 

No. 11, 1970 Bratfisch, O. , Domic, S. , and Borg, G, : PERCEIVED 
DIFFICULTY OF A MOTOR-SKILL TASK AS A FUNC- 
TION OF TRAINING 

No. 12, 1970 Hosman, J., and Borg, G. : THE METRIC STRUCTURE 
OF VERBAL EXPRESSIONS: A FURTHER INVESTI- 
GATION 

No. 13, 1970 Valter, V. : DEDUCTION AND VERIFICATION OF A 
QUANTUM PSYCHOPHYSICAL EQUATION 

No. 14, 1970 Domic, S. , Bratfisch, O. , and Santesson, A.: VERBAL 
FACTOR IN IMMEDIATE MEMORY 

No. 15, 1971 Borg, G. , Bratfisch, O. , and Dornic, S. : PERCEIVED 
DIFFICULTY OF AN IMMEDIATE MEMORY TASK 

No. 16, 1971 Borg, G. , Bratfisch, O. , Dorni^, S. : PERCEIVED 
DIFFICULTY OF A VISUAL SEARCH TASK 

No. 17, 1971 Borg, G. : A NOTE ON A "DISPERSION METHOD" IN 
PSYCHOPHYSICAL SCALING 



No. 18, 1971 Borg, G. , Edgren, B. , Marklund, G. : A SIMPLE 
WALK TEST OF PHYSICAL WORKING CAPACITY 

No. 19, 1971 Borg, G. , Cavallin, N, , EdstrSm, C. -G. , Marklund, 
G. : MOTIVATION AND PHYSICAL PERFORMANCE 

No. 20, 1971 Bratfisch, O, : A FURTHER STUDY ON SUBJECTIVE 
AND OBJECTIVE INTELLIGENCE FACTORS 

No. 21, 1971 Hosman, J. : THE EFFECT OF RANDOMIZING THE 
STARTING POINT OF CROSS -MODALITY MATCHES 

No. 22, 1971 Domic, S. , Borg, G. : VISUAL SEARCH FOR SIMPLE 
GEOMETRIC FIGURES: THE EFFECT OF TARGET- 
NOISE SIMILARITY 

No. 23, 1971 Hosman, J.: THE DISTRIBUTION OF INDIVIDUAL 
CROSS-MODALITY MATCHES 

No. 24, 1971 Borg, G, , Edstrbm, C. -G. , Marklund, G. : A 

BICYCLE ERGOMETER FOR PHYSIOLOGICAL AND 
PSYCHOLOGICAL STUDIES 

No, 25, 1972 Borg, G. , Edstrom, C. -G. , Marklund, G. : EFFECTS 
OF THE RATE OF THE WORK LOAD INCREASE ON 
TERMINAL THRESHOLDS FOR PHYSICAL WORK 

No. 26, 1972 Persaud, G. : THE PERFORMANCE OF TWO SAMPLES 
OF PRIMARY SCHOOL CHILDREN ON TWO CULTURE 
FREE AND TWO CULTURE BOUND INTELLIGENCE 
TESTS 

No. 27, 1972 Borg, G. : A RATIO SCALING METHOD FOR INTER- 
INDIVIDUAL COMPARISONS 

No. 28, 1972 Bratfisch, O. , Dorniif, S., Borg, G.: PERCEIVED 

DIFFICULTY OF ITEMS IN A TEST OF REASONING 
ABILITY 

No. 29, 1972 Bratfi-sch, O. , Borg, G., Domfd", S.: PERCEIVED 
ITEM -DIFFICULTY IN THREE TESTS OF IN- 
TELLECTUAL PERFORMANCE CAPACITY 



